1. Field of the Invention
The present invention relates to an image processing device, an image sensing apparatus, and an image processing method, which are used for carrying out image processing on an image that has been obtained from an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In particular, the present invention relates to the image quality enhancement of a shot image.
2. Description of the Related Art
An image sensor such as a CCD image sensor or a CMOS image sensor is used for an image sensing apparatus such as a digital camera or a video camera. In recent years, the number of pixels on the image sensor has been increased accompanying the reduction in the size of each pixel. The use of an image sensor having an increased number of pixels still requires high-speed processing so as not to lose usability such as when using continuous shooting speed. An exemplary method of high-speed processing includes a multi-channel approach that provides a plurality of output pathways for an image sensor so as to simultaneously read out a plurality of pixels. However, there arises a case that pixels are susceptible to being affected by noise, since optical signals become small due to the reduction in pixel size. In addition, the noise amount differs depending on the output pathways under the influence of multi-channel effects.
For example, in a multi-channeled CMOS image sensor having a plurality of output amplifiers, level differences occur in each column if there are fluctuations in the characteristics of these output amplifiers. This level difference is called “pattern noise in the vertical direction”. In addition, since the power supply and the GND (grounding) are shared for each pixel, a level difference occurs in the entire selected row if the power supply and the GND for each pixel fluctuates during the reading out of the selected row. This level difference is called “pattern noise in the horizontal direction”.
Pattern noise in the horizontal direction may cause an intensity difference in the left and right direction depending on the position at which the power supply and the GND are provided within the image sensor. Since the wiring length from the power supply and the GND for each pixel is different, an impedance of the wire is also different for different pixels. Hence, the impedance of the wiring increases the farther the pixel is positioned from the power supply and GND, whereby the influence of fluctuations in the power supply and the GND becomes stronger. As a consequence, the pattern noise intensity may increase at a position more distant from the power supply and the GND. Such pattern noise may cause image quality degradation, which requires reducing the amount of noise.
Since pattern noise in the vertical direction is to be determined uniquely by the characteristics of the output amplifiers, it can be corrected by the fluctuation correction for each output amplifier. On the other hand, pattern noise in the horizontal direction may become random if the fluctuations in the power supply and the GND are random. An exemplary method for correcting such pattern noise includes offset correction to be described below.
The image sensor is provided with an area called an optical black (hereinafter referred to as “OB”) that is light-shielded so as not to allow light reflected from an object to reach the image sensor. Pattern noise also occurs in the OB section. Thus, in the offset correction, the pattern noise amount in the horizontal direction is calculated by calculating the average for each row of the horizontal optical black (hereinafter referred to as “HOB”) provided in a strip at one end of the horizontal direction in the pixel area among the OB section. Pattern noise is corrected by subtracting this value from each row of the effective section which is non-light-shielded normal area.
Moreover, as a method for removing noise, a noise removal method utilizing frequency resolution has been proposed. Japanese Patent Laid-Open No. 2006-309749 discloses a method in which an image is resolved into low frequency components and high frequency components in a plurality of frequency bands by means of a multiple resolution transform, and noise removal processing is performed for each of the resolved low frequency components and high frequency components. In this method, an edge-preserving smoothing filter is used for noise removal processing, and the noise-removed frequency components are synthesized so as to obtain a corrected image.
In addition, Japanese Patent Laid-Open No. 2008-15741 discloses a method that repeats operations of resolving an image obtained by an image sensor into a high frequency component and a low frequency component, resolves the resulting components into a plurality of frequency bands, removes a noise component from a high frequency component in each frequency band to thereby obtain a corrected image through frequency synthesis. For noise removal from a high frequency component, a low frequency component in the same frequency band is subjected to an edge detecting filtering so as not to unintentionally remove an object edge component to thereby generate edge-preserving information. Noise from the place other than the edge section is removed utilizing this information.
Unfortunately, not only pattern noise generated for each row or column but also random noise generated randomly for each pixel are included in a shot image. When a large amount of random noise is present, the offset correction that subtracts the average for each row of the HOB area from the effective section cannot detect pattern noise accurately from the HOB area due to the influence of the random noise. In addition, when pattern noise having an intensity difference in the left and right direction occurs, the intensity difference in the left and right direction cannot be detected by the HOB area alone. As a result, in addition to a sufficient correction effect being unobtainable, the image quality may be degraded.
Since the noise removal method disclosed in Japanese Patent Laid-Open No. 2006-309749 uses the edge-preserving smoothing filter, any noise is corrected regardless of the presence or absence of noise at a place other than the edge section. Likewise, the noise removal method disclosed in Japanese Patent Laid-Open No. 2008-15741 also performs noise correction regardless of the presence or absence of noise at a position other than the position where an edge has been detected by the edge detecting filter. Therefore, conventional noise removal methods perform noise correction even at a position where there is originally no noise and thus correction is unnecessary, and the result of which may cause degradation of image resolution.